Reconciling climate - conflict meta - analyses : reply to

A comment by Buhaug et al. attributes disagreement between our recent analyses and their review articles to biased decisions in our meta-analysis and a difference of opinion regarding statistical approaches. The claim is false. Buhaug et al.’s alteration of our metaanalysis misrepresents findings in the literature, makes statistical errors, misclassifies multiple studies, makes coding errors, and suppresses the display of results that are consistent with our original analysis. We correct these mistakes and obtain findings in line with our original results, even when we use the study selection criteria proposed by Buhaug et al. We conclude that there is no evidence in the data supporting the claims raised in Buhaug et al. Buhaug et al. (2014) argue that conclusions presented in our review article (Hsiang and Burke, Climatic Change, 2014) and our reanalysis/meta-analysis of existing studies (Hsiang et al., Science, 2013a; hereafter HBM) depart from conclusions in other recent reviews because of errors in our meta-analysis. As detailed in both articles, the difference between HBM’s conclusions and earlier findings arises because other reviews omitted many existing studies and did not systematically interpret statistical uncertainty in the included studies, and HBM’s analysis had more stringent criteria for the research design and methodological rigor of included studies. Buhaug et al. suggest three reasons why HBM’s analysis is flawed and offer suggestive support for each claim. Here we examine the criticisms in turn and demonstrate that each concern arises from Buhaug et al.’s misrepresentation of HBM’s analysis, misrepresentation of prior studies in the literature, and statistical or coding errors in their alteration of the original meta-analysis. Buhaug et al.’s first criticism is that some studies in the analysis rely on similar data sets, causing the results of these related studies to potentially be correlated. If true, this would cause Climatic Change DOI 10.1007/s10584-014-1276-z This reply refers to the comment available at doi:10.1007/s10584-014-1266-1. S. M. Hsiang (*) : E. Miguel University of California, Berkeley, Berkeley, CA 94720, USA e-mail: [email protected] S. M. Hsiang : E. Miguel National Bureau of Economic Research, Cambridge, MA 02138, USA M. Burke Stanford University, Stanford, CA 94305-6055, USA the statistical uncertainty in HBM’s result to be understated, which could theoretically cause HBM’s highly statistically significant finding to be rendered insignificant. The more highly correlated data are across more studies, the larger this concern. Buhaug et al. offer one example to support their concern, a single case where data in two studies is correlated with r=0.6. But in direct contrast to Buhaug et al.’s claim that HBM ignored this issue, HBM explain it in detail and systematically test its influence on their result in Section B of HBM’s supplement, stating explicitly that “[T]he estimates of β are unlikely to be independent across all studies...” HBM find that even if they assume r=0.7 for all pairs of studies, a much more hazardous environment for HBM’s analysis than Buhaug et al.’s example of a single pair of correlated studies, HBM’s meta-analysis result remains statistically significant with 95 % confidence. Buhaug et al.’s first criticism was addressed in HBM’s original article. Buhaug et al.’s second criticism is to assert that HBM assume “causal homogeneity”, i.e., that all studies recover the same causal effect, and that this assumption “is essential for [HBM’s] meta-analysis to be meaningful.” Both assertions are false. The meta-analytic technique used in HBM explicitly assumes that effects across studies are not the same even within a given class of conflict. The Bayesian random effects approach in HBM, based on Gelman et al. (2004), allows different types of intergroup conflict in different regions to respond differently to climate variables. A central strength of HBM’s meta-analysis is to model these potential differences while simultaneously examining whether estimates across multiple studies share any common component. HBM assume neither causal homogeneity nor complete causal heterogeneity, instead allowing for any arbitrary mixture of the two and “letting the data speak”. HBM explain this approach in detail and extensively quantify and discuss the extent of heterogeneity across studies in Section B of HBM’s supplement and report in the main text, “we recover estimates for the between-study s.d. (a measure of the underlying dispersion of true effect sizes across studies) that are... two-thirds of the precision-weighted mean for intergroup conflict.... By comparison, if variation in effect sizes across studies was driven by sampling variation alone [i.e., the assumption of causal homogeneity were true], then this s.d. in the underlying distribution of effect sizes would be zero. This finding suggests that true effects probably differ across settings, and understanding this heterogeneity should be a primary goal of future research.” In direct contradiction to Buhaug et al.’s second claim, HBM do not assume causal homogeneity and instead discuss cross-study differences and formally characterize them. Buhaug et al.’s third criticism is that HBM’s results are not “balanced” because HBM use “selection criteria that explicitly disregard studies that revisit previously investigated climateconflict associations.” This is an erroneous representation of HBM’s selection method and does not accurately describe how revisited data sets were handled. Exact replications were omitted to avoid double counting but studies that revisited prior relationships were included in the review and were used to interpret findings in the prior study (see footnotes of Table 1 and Section A of the supplement in HBM). For example, follow-up analysis of Bushman et al. (2005) was used to adjust how HBM interpreted the original study by Cohn and Rotton (1997), and Buhaug (2010) which followed-up Burke et al. (2009) is fully included in the meta-analysis and discussed extensively. Buhaug et al. also criticize HBM for analyzing climate variables that are the main result of each original study, an odd criticism given that the role of meta-analysis is to summarize the main results of a collection of studies. Buhaug et al. take a different approach when altering HBM’s meta-analysis by focusing exclusively on results that researchers do not claim are important, discussed in detail below. To rigorously assess Buhaug et al.’s central claim that HBM’s result is driven by systematically biased study selection, we implement a “stress test” where we suppose that there actually are numerous “missing” results that HBM inappropriately omitted, each of which Climatic Change